We study reinforcement learning (RL) for text-based games, which are interactive simulations in the context of natural language. While different methods have been developed to represent the environment information and language actions, existing RL agents are not empowered with any reasoning capabilities to deal with textual games. In this work, we aim to conduct explicit reasoning with knowledge graphs for decision making, so that the actions of an agent are generated and supported by an interpretable inference procedure. We propose a stacked hierarchical attention mechanism to construct an explicit representation of the reasoning process by exploiting the structure of the knowledge graph. We extensively evaluate our method on a number of man-made benchmark games, and the experimental results demonstrate that our method performs better than existing text-based agents.

Figure 1 shows an example of the raw interface of the game "ztuu", where raw textual observations In this section, we show the first 15 interaction steps of two games: "zork1" and "ztuu". C h o s e n a c t i o n a n d r e w a r d A c t i o n: w e s t Reward: 0 | S c o r e: 0 ===== S t e p 2 ===== ===== 1 . C h o s e n a c t i o n a n d r e w a r d A c t i o n: s o u t h Reward: 0 | S c o r e: 0 ===== S t e p 3 ===== 16 ===== 1 . C h o s e n a c t i o n a n d r e w a r d A c t i o n: s o u t h Reward: 0 | S c o r e: 0 ===== S t e p 4 ===== ===== 1 . C h o s e n a c t i o n a n d r e w a r d A c t i o n: w e s t Reward: 0 | S c o r e: 0 ===== S t e p 5 ===== ===== 1 .

Figure 1 shows an example of the raw interface of the game "ztuu", where raw textual observations In this section, we show the first 15 interaction steps of two games: "zork1" and "ztuu". C h o s e n a c t i o n a n d r e w a r d A c t i o n: w e s t Reward: 0 | S c o r e: 0 ===== S t e p 2 ===== ===== 1 . C h o s e n a c t i o n a n d r e w a r d A c t i o n: s o u t h Reward: 0 | S c o r e: 0 ===== S t e p 3 ===== 16 ===== 1 . C h o s e n a c t i o n a n d r e w a r d A c t i o n: s o u t h Reward: 0 | S c o r e: 0 ===== S t e p 4 ===== ===== 1 . C h o s e n a c t i o n a n d r e w a r d A c t i o n: w e s t Reward: 0 | S c o r e: 0 ===== S t e p 5 ===== ===== 1 .

Strengths: This work extends the KG-A2C agent in several ways: First it uses attention to re-weight the different components of textual observations (e.g. The output of this first attention is then combined with another attention computed over multiple different sub-knowledge graphs corresponding to the connectivity of locations, objects in the current location, inventory, and anything that is connected to the current player. While none of the individual building blocks are particularly novel, the combination of all of these elements introduces a lot of flexibility to structurally decompose the different types of knowledge available in the game and allow the agent to pay attention to specific subsets of this knowledge. This flexibility pays dividends when it comes to the experimental evaluation and where this agent significantly improves on KG-A2C in nearly every game. The ablations presented validate that the full stacked architecture is indeed needed to maintain current levels of performance, and the analysis shows that the attention mechansims are working well insofar as they distribute attention correctly between locations descriptions and inventory contents as needed to generate the action.

The work is an interesting approach of extending KG-A2C with sub-graphs to achieve impressive state of the art performance on several games. Ablation studies show that this architecture is needed to achieve the performance and the attention analysis is interesting. The work could benefit from a more thorough analysis of what the model is doing (beyond just attention values which are questionable). The paper could also benefit from improved clarity in its writing.

We study reinforcement learning (RL) for text-based games, which are interactive simulations in the context of natural language. While different methods have been developed to represent the environment information and language actions, existing RL agents are not empowered with any reasoning capabilities to deal with textual games. In this work, we aim to conduct explicit reasoning with knowledge graphs for decision making, so that the actions of an agent are generated and supported by an interpretable inference procedure. We propose a stacked hierarchical attention mechanism to construct an explicit representation of the reasoning process by exploiting the structure of the knowledge graph. We extensively evaluate our method on a number of man-made benchmark games, and the experimental results demonstrate that our method performs better than existing text-based agents.